Papers with multimodal training
LaMI: Augmenting Large Language Models via Late Multi-Image Fusion (2026.acl-short)
Copied to clipboard
| Challenge: | Large Language Models lack visual grounding on visual reasoning, despite training on text alone. |
| Approach: | They propose a late multi-image fusion method that augments LLMs with test-time visual signals. |
| Outcome: | Using a late multi-image fusion method, the proposed model outperforms LLMs on visual reasoning and matches VLMs in vision-based tasks. |
What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge (2022.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation methods to measure what language models learn from multimodal training are lacking. |
| Approach: | They propose two evaluation tasks to measure commonsense knowledge in language models by using visual data to evaluate multimodal models and unimodal baselines. |
| Outcome: | The proposed evaluation tasks show that training on a visual modality improves on the visual commonsense knowledge in language models. |
The World of an Octopus: How Reporting Bias Influences a Language Model’s Perception of Color (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has raised concerns about the inherent limitations of text-only pretraining. |
| Approach: | They first generate a color dataset of human-perceived color distributions for 521 common objects and then use it to analyze and compare the color distribution found in text and the distribution captured by language models. |
| Outcome: | The proposed model improves on the CoDa color distribution, while the language model improve on the ground-truth distribution. |
Synthetic Audio Helps for Cognitive State Tasks (2025.findings-naacl)
Copied to clipboard
| Challenge: | Prior work in NLP focuses on tasks that involve extracting information about the cognitive states of human entities from text. |
| Approach: | They propose a framework for learning to add synthetic audio to text-only corpora and a system that automatically tracks audio signals to produce naturalistic audio. |
| Outcome: | The proposed framework improves on 7 cognitive state modeling tasks on text and synthetic audio data from an off-the-shelf TTS system. |
PV2TEA: Patching Visual Modality to Textual-Established Information Extraction (2023.findings-acl)
Copied to clipboard
| Challenge: | Empirical results show up to 11.74% absolute (20.97% relative) increase over unimodal baselines. |
| Approach: | They propose to patch the visual modality to the textual-established attribute in- formation extractor. |
| Outcome: | Empirical results show up to 11.74% absolute (29.9% relative) increase over unimodal baselines. |
CAPSTONE: Composable Attribute‐Prompted Scene Translation for Zero‐Shot Vision–Language Reasoning (2025.emnlp-industry)
Copied to clipboard
| Challenge: | CAPSTONE transforms visual inputs into structured text prompts that can be interpreted by a frozen Large Language Model (LLM). |
| Approach: | They propose a plug-and-play framework that transforms off-the-shelf vision models into structured text prompts that can be interpreted by a frozen Large Language Model (LLM). |
| Outcome: | The proposed framework outperforms fully trained VLMs on the POPE dataset while the 4B model achieves competitive results. |
ModRWKV: Transformer Multimodality in Linear Time (2025.emnlp-main)
Copied to clipboard
| Challenge: | Currently, multimodal studies are based on large language models with quadratic-complexity Transformer architectures. |
| Approach: | They propose a decoupled multimodal framework built upon the RWKV7 architecture as its LLM backbone and a lightweight architecture to achieve multi-source information fusion. |
| Outcome: | The proposed framework achieves multi-source information fusion through dynamically adaptable heterogeneous modality encoders. |
Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Multimodal large language models combine visual and textual data for tasks like image captioning and visual question answering. |
| Approach: | They propose temperature scaling and iterative prompt optimization to calibrate MLLMs and enhance model reliability. |
| Outcome: | The proposed techniques improve MLLMs and improve model reliability. |
How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input? (2022.coling-1)
Copied to clipboard
| Challenge: | Current language models have been criticised for learning language from text alone without connection between words and their meaning. |
| Approach: | They propose to train models on more sources than text to provide the lacking connection between words and their meanings. |
| Outcome: | The proposed model adaptation methods perform differently for different models and unimodal model counterparts perform on par with the VL models regardless of adaptation. |
LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM (2025.findings-acl)
Copied to clipboard
Sambal Shikhar, Mohammed Irfan Kurpath, Sahal Shaji Mullappilly, Jean Lahoud, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan, Hisham Cholakkal
| Challenge: | Existing speech-enabled LLMs degrade conversational quality by modifying the LLM, compromising its linguistic capabilities. |
| Approach: | They propose a lightweight 30M-parameter, LLM-agnostic, autoregressive streaming TTS system that generates high-quality speech with low latency. |
| Outcome: | The proposed system achieves a significantly lower word error rate compared to speech-enabled LLMs while operating at comparable latency. |
Grounded Concreteness: Human-Like Concreteness Sensitivity in Vision–Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | a long tradition in cognitive science treats concreteness as a graded dimension of conceptual representation . concrete words benefit from richer sensory codes and exhibit robust behavioral advantages over abstract words . |
| Approach: | They compare vision-language models with text-only large language models to test their concreteness . they find that VLMs show more human-like sensitivity to concreteness than LLMs . |
| Outcome: | The proposed model-based training improves on the Llama text backbones and Llma Vision counterparts. |